Comparing Oral Interviewing with Self-administered Computerized Questionnaires an Experiment

A previous field experiment conducted via national surveys showed that data collected via the Internet manifested higher concurrent and predictive validity and less random and systematic measurement error than data collected via telephone interviewing. To ascertain the extent to which these differences were attributable to mode per se, a laboratory experiment was conducted in which respondents were randomly assigned to answer questions either on a computer or over an intercom with an interviewer. Replicating findings from the national surveys, the laboratory experiment indicated higher concurrent validity, less survey satisficing, and less social desirability response bias in the computer mode than in the intercom mode. The mode difference in concurrent validity and non-differentiation was most pronounced among respondents with more limited cognitive skills. Taken together, these results suggest a potential inherent advantage of questionnaire selfadministration on the computer over telephone administration. As researchers are increasingly interested in conducting surveys via the Internet, it is important to understand whether shifting from oral administration of questions (in telephone or face-to-face interviews) to computer self-administered interviewing changes the answers that respondents provide. This paper reports the results of a laboratory experiment designed to assess the impact of this mode shift on survey responses. LINCHIAT CHANG is an independent contractor in San Francisco, CA, USA. JON A. KROSNICK is with Stanford University, Stanford, CA, USA and is a University Fellow at Resources for the Future. This research was funded by an Ohio State University Graduate School Alumni Research Award to Chang and was reported in a Ph.D. dissertation submitted by Chang to Ohio State University. The authors would like to thank Elizabeth Stasny, Marilynn Brewer, Ken Mulligan, and Joanne Miller for their help and advice, and Joy Baskin, Mrinalini Raina, Kameela Majied, Crystal Velazquez, Juanita Wright, and Augustina Jay for assisting in data collection. *Address correspondence to Jon A. Krosnick, Standford University, 434 McClatchy Hall, 450 Serra Mall, Standford, CA 94305, USA; e-mail: [email protected]. Public Opinion Quarterly, Vol. 74, No. 1, Spring 2010, pp. 154–167 doi: 10.1093/poq/nfp090 Advance Access publication February 12, 2010 © The Author 2010. Published by Oxford University Press on behalf of the American Association for Public Opinion Research. All rights reserved. For permissions, please e-mail: [email protected] at S taford U nirsity on A ril 9, 2010 http://poordjournals.org D ow nladed fom For this study, respondents were brought to the lab and randomly assigned to answer a questionnaire either self-administered on a computer or administered orally via an intercom with an interviewer. This design allowed us to assess whether the computer mode was associated with improved concurrent validity, less survey satisficing, and less social desirability response bias, as Chang and Krosnick (2009) found in a field experiment comparing these two modes. The experimental design also allowed us to explore whether the mode effect was moderated by respondents’ cognitive skills. Oral presentation might pose the greatest challenges for respondents with limited cognitive skills, because of the added burden imposed by having to hold a question and response choices in working memory while searching long-term memory and generating a judgment. Visual presentation of a question might reduce that burden on working memory, thereby helping people with limited cognitive skills the most. However, it may be that oral presentation makes question and response choice interpretation easier for people with limited reading skills thanwould visual presentation. If that is so, then any advantage of computer presentation might be confined to respondents high in cognitive skills and might even reverse among respondents with more limited skills. We explored these various possibilities. We also examined whether administration time varied across modes. Respondents answering questions via computer could answer questions at whatever pace was optimal for them. But the nature of oral exchange in the absence of visual cues might lead both interviewers and respondents to accelerate the pace of questioning over an intercom beyond what would be optimal. So we thought respondents in the computer mode might complete the questionnaire more slowly than those in the intercom mode.